Control device discovery in networks having separate control and forwarding devices

ABSTRACT

A Software Defined Network (SDN) includes a plurality of forwarding devices and a routing control device located separate from the forwarding devices. The routing control device, establishes paths to and from the network forwarding devices. Using such paths, forwarding devices send the routing control device information reflecting the topology if the network. Embodiments disclosed herein enable automatic discovery of the topology of the network and the paths to and from the routing control device.

BACKGROUND

1. Field

This field is generally related to network routing.

2. Related Art

A communication network may, for example, provide a network connectionthat allows data to be transferred between two geographically remotelocations. A network connection may span multiple links connectingcommunication devices such as routers. Networks may have differenttopologies depending on how the links are interconnected throughcommunication devices. Given a particular network topology, multipleroutes may be available between a source and destination. Some routesmay be more desirable than others depending on current capacity andusage.

Traditional routing algorithms rely on local information each router hasfrom its neighboring links and devices to route data. A router maintainssuch information in a routing table. And based on the destinationaddress of an incoming packet, a router uses its routing table toforward the packet to a specific neighboring device. To develop therouting table, each router uses a protocol like Border Gate Protocol(BGP) to exchange routing and reachability information with localneighboring routers. In this way, each router both forwards packets andconducts control functions to update its own routing table.

While using local information may be desirable in some contexts, it maynot always route data efficiently. To route data more efficiently,another technique, referred to as Software Defined Networks (SDNs),separates the control and forwarding functions into separate devices.The control device uses a global knowledge of the network topology todetermine a path through the network of forwarding devices forindividual data flows. In this way, the routing control device may, forexample, establish paths that minimize delay or maximize bandwidththrough the network.

BRIEF SUMMARY

In an embodiment, a computer-implemented method discovers routes fromnetwork forwarding devices to a routing control device. The discoverymethod includes receiving a control packet on a port of a forwardingdevice. The method then, based at least in part on the port thatreceived the control packet, identifies which port on the forwardingdevice to forward data to reach the routing control device. In responseto receipt of a new data stream at the forwarding device, the methodsends a command to the routing control device on the identified port.The command instructs the routing control device to determine a paththrough the forwarding devices to a destination and to configure theforwarding devices to forward the new data stream along the determinedpath. The method then forwards the new data stream according to theconfiguration commands of the routing control device.

System and computer program product embodiments are also disclosed.

Further embodiments, features, and advantages of the invention, as wellas the structure and operation of the various embodiments, are describedin detail below with reference to accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and form partof the specification, illustrate the present disclosure and, togetherwith the description, further serve to explain the principles of thedisclosure and to enable a person skilled in the relevant art to makeand use the disclosure.

FIG. 1A is a diagram that illustrates a routing control device thatsends control packets to a forwarding device.

FIG. 1B is a diagram that illustrates forwarding devices floodingcontrol packets throughout the network.

FIG. 1C is a diagram that illustrates how the control device collectsinformation reflecting the network topology.

FIG. 2 is a flowchart that illustrates a method for finding a path fromthe control device to a forwarding device.

FIG. 3 is a flowchart that illustrates a method for setting up a pathfrom each forwarding device to the control device.

FIG. 4A is a diagram that illustrates a network connection using a pathfrom a forwarding device to the control device.

FIG. 4B is a diagram that illustrates the routing control device settingup a fast path.

FIG. 5 is a diagram that illustrates a network with multiple controldevices.

FIG. 6 is a diagram that illustrates modules of a control device and aforwarding device.

The drawing in which an element first appears is typically indicated bythe leftmost digit or digits in the corresponding reference number. Inthe drawings, like reference numbers may indicate identical orfunctionally similar elements.

DETAILED DESCRIPTION

As mentioned above, SDN routing techniques use global knowledge of thenetwork topology to route packets efficiently. These techniques use arouting control device and forwarding devices separate from each other.When a forwarding device receives a new data flow through a network, theforwarding device contacts the routing control device to determine a newpath for the data flow. The routing control device determines the newpath through the network's forwarding devices and configures theforwarding devices accordingly.

To operate, the routing control device and forwarding devices need to beconfigured with network information. In particular, to request that anew path be created through the interconnected forwarding devices, theforwarding devices need to know how to route data to the routing controldevice. And, the routing control device needs to know the networktopology to determine a path through the network. Manually configuringthis information can be time-consuming and error-prone.

Embodiments disclosed here enable topology information to be discoveredautomatically. In an embodiment, the routing control device is connectedto a forwarding device and sends control packets to it. The controlpacket contains the address of the control device as its source addressin an address field. The control device also attaches a serial number toeach control packet.

The forwarding device attaches performance information that indicates,for example, how long it would take for the network to send informationfrom the forwarding device to the routing control device. Then, theforwarding device broadcasts, or floods, the control packet on its otherports, which are connected to other forwarding devices. And, if theother forwarding devices have not previously rebroadcasted a packet withbetter performance information, they update the performance informationand re-broadcast the packet.

As the forwarding devices receive the control packets, they rememberwhich port the control packet received the packet with the bestperformance information. This port, the forwarding device knows, is onthe fastest route to the routing control device. And, each forwardingdevice can request that a label-switched path be set up on that route tothe routing control device.

With the paths established, the forwarding devices send informationconcerning their neighborhood network topology and network condition tothe control device. Using such information from all forwarding devices,the control device can create a database of global network topology andnetwork condition. This topology database enables the control device toestablish a path between a sources forwarding device and a destinationforwarding device in the network.

Once this path is established to the controller, the controller may, atany later time, determine a better path for the communications betweenthe forwarding device and the controller using another method which mayutilize more complete topology and performance information. If thisbetter path is determined, the existing path may be replaced by thecontroller resignalling the specific new path and then eliminating theexisting path.

The Detailed Description that follows is divided into five sections. Thefirst section describes, with respect to FIGS. 1A-1C, a control devicein the network sending control packets to each forwarding device, andgathering topology and network condition information from the forwardingdevices. The second section describes, with respect to FIGS. 2-3,setting up bidirectional connections between the control device and eachforwarding device. The third section describes, with respect to FIGS.4A-B, using the control device to establish and transfer data withoutrequiring edge routers to buffer a large amount of packets. The fourthsection describes, with respect to FIG. 5, using multiple controldevices in the network. The fifth and final section describes, withrespect to FIG. 6, a control device system and its modules and aforwarding device system and its modules.

Control Device and Control Packets

FIG. 1A is a diagram 100 illustrating a communication network. Thecommunication network may be a local area network (LAN), metropolitanarea network (MAN), or wide area network (WAN). It may utilize anypoint-to-point or multipoint-to-multipoint networking protocols. Thenetwork access protocols used may include, for example, Multi ProtocolLabel Switching (MPLS), Ethernet, Asynchronous Transfer Mode (ATM),High-Level Data Link Control (HDLC), or Packet Relay.

The communication network includes a plurality of forwarding devices,such as forwarding devices 104, 106, 108, 110, and 112, interconnectedby links. Forwarding devices are devices that forward packets, includingdevices at the data link layer (OSI layer 2) and the network layer (OSIlayer 3).

The communication network also includes a routing control device 102.Routing control device 102 may be connected to at least one forwardingdevice, for example forwarding device 104. Routing control device 102may be geographically remote from other forwarding devices in thenetwork.

In an example, a user may send data from a source forwarding device, forexample forwarding device 106, to a destination forwarding device, forexample 110. The data may be a stream divided into packets, and eachpacket may specify forwarding device 110, or another downstream device,as its destination.

Routing control device 102 provides intelligent routing for establishinga network connection. To do so, the control device 102 requiresknowledge of the topology and the condition of the network links anddevices. Routing control device 102 is directly connected a forwardingdevice 104. For the control device to gain knowledge of the network, thecontrol device 102 sends a control packet 120 to the forwarding device104.

Control packet 120 may include an indication that the packet is acontrol packet, an identifier of routing control device 102, such as itsMedia Access Control address, and a serial number of the packet. In anembodiment the control device 102 may send a new control packet atuniform time intervals. Each time control device 102 sends a new packet,control device 102 may increment the serial number, sending the newcontrol packet out with a different number.

In an illustrative example, control packet 120 may include the followinginformation:

Type Flag: C (for control packet)

MAC Address of Control Device: 01:23:45:67:89:ab

Serial No. of Control Packet: 1

Once routing control device 102 sends the control packet 120 to theadjacent forwarding device 104, the network forwarding devices flood thepacket across the network as illustrated in FIG. 1B.

FIG. 1B is a diagram illustrating the network forwarding devicesflooding a control packet across the network. After receiving controlpacket 120, the forwarding device 104 may send out the control packet onall its ports other than the port that received it.

As mentioned above, when routing control device 102 generates controlpacket 120, the control device may include its address and a serialnumber in the control packet. As forwarding device 104 receives thecontrol packet, and before forwarding it, the forwarding device 104 maymake two modifications. First, it may add its own identifier (such asits MAC address) to the control packet. This address is added to anaddress field in the control packet for keeping track of all theforwarding devices the control packet travels through. Second,forwarding device 104 may modify a performance metric in the controlpacket. The performance metric can include, for example, the delay insending the control packet from control device 102 to forwarding device104. This delay information may be collected, for example, by forwardingdevice 104 using Link Layer Discovery Protocol (LLDP) packet exchanges.The delay may be determined as a moving average or otherwise smoothedout to prevent volatile changes as the network reacts to changes intraffic volume.

With those two modifications, forwarding device 104 forwards the controlpacket on its other ports as control packets 142, 144, and 146. At thispoint, all three control packets 142, 144, and 146 may contain the sameinformation. Continuing the example above and assuming the MAC addressof forwarding device 104 is 02:02:02:02:02:bc and the delay betweencontrol device 102 and forwarding device 104 is 5 ns, the controlpackets 142, 144, and 146 may include the following information:

Type Flag: C (for control packet)

MAC Address of Control Device: 01:23:45:67:89:ab

Serial No. of Control Packet: 1

Addresses of Intermediate Forwarding Devices: 02:02:02:02:02:bc

Performance Metric: 5 ns

Forwarding device 104 forwards control packet 144 on link 114 toforwarding device 106; control packet 142 on link 116 to forwardingdevice 108; and control packet 146 on link 118 to forwarding device 110.

Once forwarding device 106 receives the control packet 144, it adds itsown address to the address field and updates the performance metric ofthe control packet to generate a control packet 148. For example,forwarding device 106 may add the delay to transmit data from forwardingdevice 106 to forwarding device 104. Continuing the example above andassuming the MAC address of forwarding device 106 is 03:03:03:03:03:cdand the delay between controller 102 and forwarding device 104 is 2 ns,the control packet 148 may include the following information:

Type Flag: C (for control packet)

MAC Address of Control Device: 01:23:45:67:89:ab

Serial No. of Control Packet: 1

Addresses of Intermediate Forwarding Devices: 02:02:02:02:02:bc,03:03:03:03:03:cd

Performance Metric: 7 ns

Forwarding device 106 transmits packet 148 on link 122 to forwardingdevice 108.

Forwarding device 108 receives control packets 142 and 144. Using thedelay information in each of these two control packets, forwardingdevice 108 can choose to establish a path with less delay to the controldevice 102. Using the address sequence field in the control packet,forwarding device 108 can know to get to the control device, whichneighboring forwarding device it needs to communication with.

For example, forwarding device 108 may know that the time to transmitdata to forwarding device 106 on link 120 is 3 ns, and that the time totransmit data to forwarding device 104 on link 116 is 9 ns. Adding thatto the performance metrics received in packet 148 and 142, forwardingdevice 108 can determine that transmission to control device 102 viaforwarding device 106 takes 10 ns, while transmission via forwardingdevice 104 takes 14 ns. For that reason, forwarding device 108 maychoose to route data to control device 102 via forwarding device 106,and not device 104, taking the faster path.

Also, forwarding device 108 may reforward control packets 142 and 148.If forwarding device 108 receives the packet denoting the faster path—inthis case, packet 148—before the packet denoting the slower path—in thiscase, packet 142—the forwarding device 108 may not forward the slowerpacket. This is because forwarding device 108 knows that any data headedto control device 102 should take the faster path through forwardingdevice 106, regardless of whether it originated from forwarding device108 or another forwarding device, such as device 112.

Having determined the best path to control device 102 is throughforwarding device 106, forwarding device 108 may establish abi-directional path with forwarding device 106. In an embodiment, toestablish this bi-directional path, forwarding device 108 sets a rule inits routing table, which also may be referred to a forwarding table,that in order to get to the control device, it sends a packet on aspecific port (the port that it received control packet 144) and with aparticular label. This information is communicated with forwardingdevice 106, so both forwarding devices follow the same rule. Therefore,when the forwarding device 106 receives the packet from forwardingdevice 108 with the set label and on the set port, it knows that thispacket is intended for control device 102. Similarly, forwarding device106 knows to forward traffic for control device 102 through forwardingdevice 104. So, forwarding device 106 sets up a label with forwardingdevice 104 to route data to control device 102. Labels may also be setup to route data from control device 102 through forwarding device 106to forwarding device 104 in a similar manner.

In this way, by analyzing and reforwarding control packets from acontrol device, each forwarding device can discover how to communicatewith the control device and set up a label-switched path to the controldevice. As described below with respect to FIGS. 4A-B when theforwarding device receives a new data stream, the forwarding device mayforward packets from the new data stream to the control device until thecontrol device sets up a path for the packets through the network.

In addition to using the label-switched path to forward data to thecontrol device, embodiments may also use the label-switched path to helpthe control device discover the topology information of the network asillustrated in FIG. 1C.

FIG. 1C is a diagram 170 that illustrates control device 102 collectinginformation reflecting the network topology.

Each forwarding device sends information to the routing control devicethat shows what forwarding devices it is connected to, and what theconditions of the surrounding links and forwarding devices are.Conditions of surrounding links and forwarding devices can include, forexample, traffic and congestion information, latency, packet loss rate,jitter, etc. This information may be collected, for example, using LinkLayer Discovery Protocol (LLDP). In an embodiment, the forwardingdevices send this information periodically. And, the forwarding devicesuse the paths established between them and the control device to sendthis information.

In the example shown in FIG. 1C, a path for forwarding device 108 to thecontrol device 102 is established. This path includes forwarding devices108, 106, and 104. Forwarding device 108 sends the information includingits surrounding topology to the control device 102. In an example,forwarding device 108 sends this information in a packet 172 to routingcontrol device 102.

Routing control device 102, using the topology information it receivesfrom all of the forwarding devices including forwarding device 108,creates a database 174 that reflects the topology of the network of theforwarding devices.

Establishing Paths to and from the Control Device

FIG. 2 is a flowchart that illustrates a method 200 for finding a pathfrom the control device to a forwarding device.

At step 202, the control device physically connects to a forwardingdevice of the network of forwarding devices. At step 204, the controldevice sends a control (or discovery) packet, for example a Link LayerDiscovery Protocol (LLDP) packet, to the forwarding device it isconnected to.

The forwarding device receiving the packet, at step 206 verifies whetherthe packet originated from a control device. In an example, theforwarding device can verify that the packet is from a control deviceby, for example, checking to see if a particular flag is raised.

The control packet includes a performance metric that shows conditionsof the network links or network devices. For example the metric can showlink latency. Therefore, in this example, a lower metric shows a lowerlatency therefore a better performance. The metric can also be afunction of multiple conditions such as latency, packet loss, jitteretc.

Each forwarding device that receives the control packet, updates theperformance metric. The updated metric, for example, shows theadditional latency for the control packet to reach the forwarding devicefrom a previous, or upstream, forwarding device. After updating themetric, the forwarding device floods the control packet on all its portsother than the port where the control packet was received.

A forwarding device may receive again the same control packet that ithad received before. This may happen because the control packets areflooded through the network. The forwarding device can recognize therepeat control packet by the unique serial number that the controldevice attaches to each control packet.

The forwarding device, at step 208, checks to see if packet'sperformance metric is better than previous metrics received from thecontrol device. If the metric is the best performance metric among allcontrol packets received by the forwarding device, at step 210, theforwarding device stores packet's incoming port as the best way to reachthe control device.

At step 212, the forwarding device checks the serial number to determinewhether the control packet has been previously forwarded, to avoid acontrol packet from indefinitely circulating through the network. Sincethe control packet adds a unique serial number to each control packet itgenerates, the forwarding devices can use the serial number of a controlpacket to determine whether it has previously forwarded the controlpacket or not. Or, the forwarding devices can use the serial number todetermine whether the current control packet is out of date if, forexample, they had previously forwarded a more recent control packet witha greater serial number. A number of other mechanisms may be used toavoid indefinitely circulating packets through the network such ashashes or checksums of at least a portion of the control packet.

The forwarding device, at step 214, determines whether the packet hasnot previously been forwarded and includes a metric showing the bestperformance. If either the control packet does not show the bestperformance metric, or it has been previously forwarded by theforwarding device, the forwarding device discards the packet and theprocess ends.

If the control packet contains the best performance metric and is notpreviously forwarded, the process 200 sets up path at step 216. Step 216is described in details in the description of FIG. 3 below.

Next, at step 218, the forwarding device updates the performance metricto reflect the performance of the link connecting the upstreamforwarding device and the current forwarding device, and the performanceof the two forwarding devices themselves. Next, at step 220, thereceiving forwarding device floods the packet on all its ports, otherthan the port where it received the control packet. In this way, theupdated control pack is sent to other forwarding devices.

The other forwarding devices repeat the process from step 206,processing the packet in the same manner.

FIG. 3 is a flowchart that illustrates a method 216 for setting up apath between forwarding devices 104 and 106. As an example, if a controlpacket has traveled to forwarding device 106, from the forwarding device104, forwarding device 104 is an upstream forwarding device for 106.

Continuing the above with respect to FIGS. 1A-B and 2, assume forwardingdevice 106 receives from the upstream forwarding device 104 a newcontrol packet and the new control packet has the best performancecharacteristics to date. For this reason, a path is set up at step 216in FIG. 2.

At step 302, forwarding device 106 sends request to the next upstreamforwarding device for a path to the control device. Forwarding device104 at step 306 determines a label for establishing the path. Forwardingdevice 104 sends this label to forwarding device 106, at step 308.Forwarding device 106 establishes this label rule accordingly in itsrouting table at step 304.

At step 310, forwarding device 104 requests path to control device. Tocreate a path, forwarding device 104 may request a label from anotherupstream forwarding device, repeating steps 302, 306, 308, and 310.Forwarding device 104 establishes label rule in its routing table atstep 312.

For example, forwarding device 104, after receiving the request fromforwarding device 106 at step 302, chooses label L1 for the path betweenthe two. After sending this label to forwarding device 106, forwardingdevice 106 will enclose this label on all the data packets destined tothe control device.

In this example, forwarding device 104, upon receiving data packets withlabel L1 on the port connected to the forwarding device 106, forwardsthe packets to the control device. To forward packets to the controldevice, forwarding device 104 may know (having established a label ruleat step 312) to forward on a particular port with a particular label. Inthis way a path from the forwarding device to the control device isestablished.

Also, a path may be established in the other direction: from the controldevice to the forwarding device in a similar manner. In that case, eachforwarding device may set up a label in its routing table. The entry inthe routing table may include the associated port to route the data to.

After such paths are established among forwarding devices in thenetwork, a path between the control device to and from each forwardingdevice in the network exists. In the example of FIG. 3, after the pathis established between forwarding device 106, forwarding device 104, andthe control device, forwarding device 106 uses this path to send data tothe control device. And vice versa, the control device uses this pathfor sending data to the forwarding device 106.

Establishing Fast Paths for New Data Streams

FIG. 4A shows a diagram 400 illustrating a data stream from network user442 addressed to server 444. The data stream includes packets 402, 404,406, 408, 410, and 412. When the first packet of the data stream—packet402—reaches forwarding device 106, it is routed to control device 102.Control device 102 determines that this packet belongs to a newconnection and starts the process of establishing a fast path for theconnection, which does not flow through the control device 102.

In the example, after packet 402, packets 404 and 406 are also sent.Forwarding device 106 may buffer packets 404 and 406 or may forwardsthose packets onto control device 102, just as it did with the firstpacket 402.

After the control device receives the first packet, it sends the packetto forwarding device 104. Forwarding device 104, using the destinationaddress included in the packet, and the paths set up at step 216 inFIGS. 2 and 3, forwards the packet to the corresponding port with thecorresponding label.

Next, the control device computes a path from the source to thedestination. The path comprises network links and routers, but not thecontrol device. For example, using the MPLS protocol, the control devicecreates a label-switched path (LSP) for the data stream. To determinethe path, the control device may take into account current or historicalbandwidth usage. For example, the control device may consider networkusage during previous weeks (or another cyclical period). The controldevice may also create multiple paths for the connection to enable loadbalancing on multiple paths.

The control device may also take into account latency, jitter, packetloss, or any other performance metric across various paths, the user'sservice level agreement, or the type of data being transferred. Forexample, broadcast video data may require a great deal of bandwidth, butlatency may be relatively unimportant. Voice-over-IP (VoIP) data, on theother hand, may not require as much bandwidth, but latency may be moreimportant. For broadcast video data, the control device could select ahigh-bandwidth, high-latency path, and, for VoIP data, the controldevice could select a low-bandwidth, low-latency path.

In another embodiment, the control device may route data through aparticular server. For example, data having a certain type or directedto a certain destination may need to be scrubbed by a particularscrubbing server en route. The scrubbing server may be used to scan thedata for malicious content, to monitor incoming data, or to performother analysis on the data. In that embodiment, the control device maydetermine such that it goes through the particular scrubbing server orset of servers.

After determining the fast path, the control device updates routingtables for each forwarding device along the path. If multiple paths arecomputed for the data stream, the control device creates routing tablesin accordance with all the paths. In one embodiment, the updated routingtables may instruct the forwarding device how to forward packets havinga particular combination of source/destination addresses andsource/destination ports. In an alternative embodiment, the data streammay be identified with a label and the updated routing table mayindicate how to forward packets having the label.

The control device sends updated routing tables to network routers. Thecontrol device can use the paths established to all the forwardingdevices in step 216 to send the routing tables. An example of how therouters are configured is illustrated in FIG. 4B.

FIG. 4B shows a diagram 420 illustrating how control device 102configures the network forwarding devices to establish the path betweenuser 442 and server 444.

In FIG. 4B, control device 102 determines that the data stream betweenuser 442 and server 444 follows the path including links 120, 122, 124and forwarding devices 106, 108, 112, and 110. To configure theforwarding devices, the control device 102 sends the updated routingtables using the configuring commands 422, 424, 426, and 428. Theseconfiguring commands configure all the forwarding devices on the fastpath connecting user 442 to destination 444. In particular, command 422instructs forwarding device 106 to forward packets in the data stream tolink 120; command 424 instructs forwarding device 108 to forward packetsin the data stream to link 122; command 426 instructs forwarding device112 to forward packets in the data stream to link 124; and command 428instructs forwarding device 110 to forward packets in the data stream tolink 124.

In an embodiment, to ensure that packets continue to flow through thecontrol device until the path is fully established, routers along thepath are configured from the egress point (forwarding device 104) to theingress point (forwarding device 106). First, command 428 configuresforwarding device 110. Second, command 426 configures forwarding device112. Third, command 424 configures forwarding device 108. Fourth,command 422 configures forwarding device 106.

Referring back to FIG. 4A, until the path is established, user 442continues to transmit packets of the data stream—packets 402, 404, and406. As mentioned above, forwarding device 106 continues to direct thesepackets on the path to control device 102.

On receipt of these packets, control device 102 routes them to theirdestination using the previously determined default path at step 216 inFIG. 2. In this way, while the path through the network is beingestablished (as illustrated in FIG. 4B), packets continue to be routedto their destination, avoiding the need to buffer the initial packets inthe edge router.

Once the path is set up (e.g., the forwarding devices are configuredwith their new routing tables), data flows along the fast path set up bythe control device. At this point, data may flow at a greater ratethrough the forwarding devices and lower end to end latency, since it nolonger goes through the control device.

Multiple Control Devices

FIG. 5 is a diagram that illustrates a network with multiple controldevices. In an example embodiment shown in FIG. 5, control device 102 isconnected to forwarding device 104, control device 504 is connected toforwarding device 110, and control device 506 is connected to forwardingdevice 506.

In the example embodiment, each control device 102, 504 or 506 sendscontrol packets, periodically or at different time intervals to theforwarding device directly connected to it. And following the sameprocess as illustrated in FIG. 2, each control device establishestwo-directional paths to each forwarding device in the network.

Each forwarding device, in the example embodiment of FIG. 5, sendsinformation showing the topology of its neighboring forwarding devicesand the network condition information to each control device in thenetwork. In the embodiment, forwarding devices use the bi-directionalpaths established at step 216 of process 200, to send this information.In an embodiment, forwarding devices send this information to controldevices periodically or at different time intervals.

A single physical network may be mirrored on multiple control deviceswith multiple topology databases for redundancy purposes. Each of thesecontrol devices has paths established between themselves and each routerin the network. In the event that a default path to one control deviceis failed, traffic may choose a path to a different control device.

In an example embodiment, each forwarding device picks one controldevice at a time for forwarding data and establishing a fast path to adestination. In the embodiment, the forwarding device can pick any oneof the control devices based on a choosing algorithm. The choosingalgorithm can be, for example, a round robin algorithm. The forwardingdevice may also consider network load or number of new connections acontrol device is setting up at a time in the algorithm for choosing acontrol device. In addition, the forwarding device may ask thecontroller with the lowest latency to establish a fast path to adestination. The round robin approach may be used in conjunction withthe lowest-latency approach. For example, a device may choose to balancetraffic across multiple paths to the controllers, perhaps by roundingthe metric to a closest 5 for example, therefore treating performancevalues of 3, 4, 5, 6, 7 as the same as a value of 5. Among thosecontrollers with the lowest rounded metric, the forwarding device mayadopt a round robin algorithm to select the controller to create thefast path. This helps to limit the impact of slight modifications in theperformance values as the network reacts to real world situations.

In an embodiment, control devices in the network need not besynchronized with one another. If a control device fails, after notreceiving control packets from that control device for a period of time,forwarding devices remove the labels for maintaining a path to thatcontrol device. Forwarding devices continue using other control deviceswith the same choosing algorithm such as round robin.

Having multiple control devices highly increases the resiliency of thenetwork. If a control device fails, or a link connecting to a controldevice fails, forwarding devices can continue using other controldevices in the network. Multiple control devices also help balancingload in the network. By intelligently choosing a control device that hasa lesser number of new connection requests, or by choosing a controldevice with the best path to, such as the path with lowest latency,forwarding devices can avoid congested segments of the network whenrequesting a new connection.

Control Device and Forwarding Device Modules

FIG. 6 shows a diagram 600 illustrating control device 102 and itsmodules and forwarding device 104 and its modules.

In particular, control device 102 includes a controller discovery module610, a path determination module 608, a data forward module 612, and atopology database 606. The various modules may operate as describedabove with respect to the method in FIG. 2.

Controller discovery module 610 is configured to determine a topology ofthe network of the forwarding devices. The controller discovery module610 sends control packets to the forwarding device directly attached tothe control device. The control packets include the address of thecontrol device and a serial number. Using the combination of the addressand the serial number each control packet can be uniquely identifiedthroughout the network.

As control packets travel through the network, each forwarding devicereceiving a control packet adds its own address to the control packet.Also, each forwarding device updates the metric showing networkcharacteristics to each control packet. Upon receiving a control packet,a forwarding device knows the path the control packet has traveled toarrive at the forwarding device from the control packet. Also using theperformance metric information, the forwarding device knows which path,among multiple paths traveled by multiple forwarding packets, is thebest path to the control device, as described by process 200. Thereforeforwarding devices can dynamically update the best path to the controldevice.

After a path to a forwarding device is set up, controller discoverymodule 610 receives topology and network condition information from theforwarding devices in the network. Using this information and similarinformation from other forwarding devices in the network, controllerdiscovery module 610 creates a topology database 606.

Topology database 606 stores the discovered topology of the forwardingdevices in the network. Topology database 606 reflects the currenttopology of how forwarding devices and links are connected together.Also, topology database 606 stores information showing the currentcondition of the network. Condition of the network can include latency,packet loss, congestion or jitter on network links or devices.

Path determination module 608 establishes paths for new networkconnections. Upon a request for connection from a source forwardingdevice, path determination module 608 determines a connection path fromthe source forwarding device to a destination forwarding device. Todetermine a path, path determination module 608 uses topology database606. Using the topology and network condition information stored in thetopology database 606, path determination module 608 can find an optimumpath from the source forwarding device to a destination. The optimumpath can, for example, be a shortest path, or a path with least delay,or a path with least packet loss.

After determining that path for a connection, path determination module608 also sends configuring commands to each forwarding device on thedetermined path. As illustrated in the example embodiment in FIG. 4B,path determination module creates and sends the configuring commands422, 424, 426, and 428. The configuring commands configure thecorresponding forwarding devices. The configured forwarding devicesroute packets belonging to the connection on the path determined for theconnection by path determination module 608.

Data forward module 612 carries the initial packets for the requestedconnection while path determination modules determine the path for theconnection and configures the forwarding device by sending configuringcommands. As illustrated in FIG. 4A, the initial packets 402, 404, 406,408, and 410 are forwarded to the control module over the path from thesource forwarding device to the control device. Data forwarding module612 forwards the initial packets over the path from the control deviceto the destination forwarding device, while the connection path is beingestablished.

In the example embodiment shown in FIG. 6, forwarding device 104includes a path request module 622, a switch discovery module 624, arouting table 626, and a performance metric module 628.

Path request module 622, after receiving a request for a new dataconnection at the forwarding device, sends a message to the routingcontrol device requesting establishing a connection path. The pathrequest module 622 uses the path to the control device, set up at step216 of process 200, to send the request for establishing connectionpath. The path request instructs the routing control device to determinea path from the source forwarding device to a destination forwardingdevice. The path determination module 608 then determines the connectionpath and configures the forwarding devices to forward the new datastream along the determined path.

Switch discovery module 624 receives the control packets sent by thecontroller discovery module 610 of the control device. Switch discoverymodule 624 determines a path from the forwarding device to the controldevice as shown in step 216 of method 200. In an embodiment, switchdiscovery module 624 makes this determination based on the informationin the control packet and the port on which it receives the controlpacket.

In order to establish the path from the forwarding device to the controldevice, switch discovery module 624 identifies, based in part on theport that received the control packet, which port on the forwardingdevice to forward data to reach the routing control device.

In an embodiment, if the performance metric in the control packetarriving from a neighboring device shows a path from the control deviceto the forwarding device with best performance, switch discovery module624 determines the label to route to control device, as shown in step306 of method 300.

Switch discovery module 624 also, after determining whether a receivedcontrol packet indicates a best performance path and is not alreadyforwarded on other ports of the forwarding device, as shown in step 214of process 200, floods the control packet on its ports other that theport where it has received the control packet.

Routing table 626 of a forwarding device contains information on how toforward a data packet in the network. Configuring commands 422, 424,426, and 428 update the routing table in the forwarding devices withinformation on how to forward a packet. Also after receiving a label forsetting up a path to control device 102, the switch discovery module 624establishes forwarding rules in the routing table 626 based on thedetermined labels.

Performance metric module 628 updates the performance metric field of anincoming control packet. An incoming control packet includes a fieldthat contains a network performance metric of the path from the controldevice up to the previous forwarding device. The performance metricmodule updates the performance metric field so that it reflects theperformance of the path up to the current forwarding device.

Performance metric module 628 also determines whether the forwardingdevice has previously forwarded another control packet, with a betterperformance metric. Performance metric module 628 informs the switchdiscovery module 624 of whether a previous same control packet with abetter metric has been flooded on the forwarding device ports.

CONCLUSION

The term “user,” as used herein, may encompass both a customer of thenetwork connectivity service, such as an employee of a business thatutilizes the network connectivity service, and a network administratorof the service provider itself. Users may also be at different companiesor organizations.

Topology database 606 may be any stored type of structured memory,including a persistent memory. In examples, each database may beimplemented as a relational database or file system.

Each of the devices and modules in FIG. 6 may be implemented inhardware, software, firmware, or any combination thereof.

Each of the devices and modules in FIG. 6 may be implemented on the sameor different computing devices. Such computing devices can include, butare not limited to, a personal computer, a mobile device such as amobile phone, workstation, embedded system, game console, television,set-top box, or any other computing device. Further, a computing devicecan include, but is not limited to, a device having a processor andmemory, including a non-transitory memory, for executing and storinginstructions. The memory may tangibly embody the data and programinstructions. Software may include one or more applications and anoperating system. Hardware can include, but is not limited to, aprocessor, a memory, and a graphical user interface display. Thecomputing device may also have multiple processors and multiple sharedor separate memory components. For example, the computing device may bea part of or the entirety of a clustered or distributed computingenvironment or server farm.

Identifiers, such as “(a),” “(b),” “(i),” “(ii),” etc., are sometimesused for different elements or steps. These identifiers are used forclarity and do not necessarily designate an order for the elements orsteps.

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

What is claimed is:
 1. A method for discovering routes from a pluralityof forwarding devices to a routing control device, comprising: (a)receiving, on a data link layer port of a forwarding device of theplurality of forwarding devices, a control packet originating from therouting control device, wherein the data link layer port connects theforwarding device to another forwarding device via a link, and thecontrol packet includes a performance metric indicating an estimatedtime to route a packet from the other forwarding device to the routingcontrol device; (b) identifying, based at least in part on (i) the datalink layer port that received the control packet and (ii) theperformance metric in the control packet, which data link layer port,identified by a MAC address, on the forwarding device to forward data toreach the routing control device; (c) in response to receipt of a newdata stream at the forwarding device, sending a message to the routingcontrol device on the data link layer port identified in (b), themessage instructing the routing control device to determine a paththrough the plurality of forwarding devices to a destination and toconfigure a subset of the plurality of forwarding devices to forward thenew data stream along the determined path; and (d) forwarding the newdata stream according to configuration commands of the routing controldevice.
 2. The method of claim 1, wherein the sending (c) comprisesforwarding the initial packet of the new data stream through the controldevice.
 3. The method of claim 1, further comprising (e) forwarding,from the forwarding device, the control packet to another forwardingdevice on a remaining data link layer port of the forwarding devicedifferent from the data link layer port that the packet was received onin (a).
 4. The method of claim 3, further comprising: (f) at theforwarding device, sending a label to another forwarding device in theplurality of forwarding devices, wherein the label, when attached to apacket received at the forwarding device indicates to the forwardingdevice to forward the packet toward the routing control device on thedata link layer port identified in (b).
 5. The method of claim 3,wherein the control packet includes a serial number, and furthercomprising: (f) determining whether the forwarding device has previouslyforwarded another control packet that is from the routing controldevice, has the serial number, and indicates another estimated time toroute a packet from the forwarding device to the routing control devicelower than the estimated time in the control packet, and wherein thestep forwarding (e) occurs when the forwarding device is determined tohave not previously forwarded the other control packet in (f).
 6. Themethod of claim 1, further comprising: (e) sending a message to therouting control device on the data link layer port identified in (b),the message identifying the forwarding device and other forwardingdevices in the plurality of forwarding devices that the forwardingdevice is connected to, wherein the routing control device uses themessage to build a database representing a topology of the network anduses the database to determine the path.
 7. The method of claim 1,wherein the control packet includes an identifier of the routing controldevice, and further comprising: (e) receiving, on a data link layer portof the forwarding device, another control packet including an identifierof another routing control device; (f) identifying, based at least inpart on the data link layer port that received the other control packet,which data link layer port on the forwarding device to forward data toreach the other routing control device; and (g) in response to receiptof a new data stream at the forwarding device, determining which of therouting control device and the other routing control device to send themessage instructing that the path be configured through the plurality offorwarding devices.
 8. A forwarding device for discovering routes to acontrol device, comprising: a switch discovery module configured to: (i)receive, on a data link layer port of the forwarding device, a controlpacket originating from the routing control device, wherein the portconnects the forwarding device to another forwarding device via a linkand the control packet includes a performance metric indicating anestimated time to route a packet from the other forwarding device to therouting control device, and (ii) identify, based at least in part on (i)the data link layer port that received the control packet and (ii) theperformance metric in the control packet, which data link layer port,identified by a MAC address, on the forwarding device to forward data toreach the routing control device; a path request module configured to,in response to receipt of a new data stream at the forwarding device,send a message to the routing control device on the identified data linklayer port, the message instructing the routing control device to: (i)determine a path through the plurality of forwarding devices to adestination and (ii) configure a subset of the plurality of forwardingdevices to forward the new data stream along the determined path; and arouting table within the forwarding device populated to enable theforwarding device to forward the new data stream according toconfiguration commands of the routing control device.
 9. The device ofclaim 8, wherein the path request module is further configured toforward the initial packet of the new data stream through the routingcontrol device.
 10. The device of claim 8, wherein the switch discoverymodule is configured to forward, from the forwarding device, the controlpacket to another forwarding device on a remaining data link layer portof the forwarding device different from the data link layer port thatthe control packet was received on.
 11. The device of claim 10, whereinthe switch discovery module is configured to send a label to anotherforwarding device in the plurality of forwarding devices, wherein thelabel, when attached to a packet received at the forwarding device,indicates to the forwarding device to forward the packet toward therouting control device on the data link layer port identified by theswitch discovery module.
 12. The device of claim 10, wherein the controlpacket includes a serial number, and the forwarding device furthercomprises a performance metric module configured to: determine whetherthe forwarding device has previously forwarded another control packetthat is from the routing control device, has the serial number, andindicates another performance metric of routing a packet from theforwarding device to the routing control device better than theperformance metric in the control packet, and wherein the forwardingoccurs when the path request module determines to have not previouslyforwarded the other control packet.
 13. The device of claim 10, whereinthe switch discovery module is further configured to: send a message tothe routing control device on the identified data link layer port, themessage identifying the forwarding device and other forwarding devicesin the plurality of forwarding devices that the forwarding device isconnected to, wherein the routing control device is configured to usethe message to build a database representing a topology of the networkand to use the database to determine the path.
 14. The device of claim10, wherein the control packet includes an identifier of the routingcontrol device, and the switch discovery module is further configuredto: receive, on a data link layer port of the forwarding device, anothercontrol packet including an identifier of another routing controldevice; identify, based at least in part on the data link layer portthat received the other control packet, which data link layer port onthe forwarding device to forward data to reach the other routing controldevice; and in response to receipt of a new data stream at theforwarding device, determine which of the routing control device and theother routing control device to send the message instructing that thepath be configured through the plurality of forwarding devices.
 15. Aprogram storage device tangibly embodying a program of instructionsexecutable by at least one machine to perform a method for discoveringroutes from a plurality of forwarding devices to a routing controldevice, the method comprising: (a) receiving, on a data link layer portof a forwarding device of the plurality of forwarding devices, a controlpacket originating from the routing control device, wherein the datalink layer port connects the forwarding device to another forwardingdevice via a link, and the control packet includes a performance metricindicating an estimated time to route a packet from the other forwardingdevice to the routing control device; (b) identifying, based at least inpart on (i) the data link layer port that received the control packetand (ii) the performance metric in the control packet, which data linklayer port, identified by a MAC address, on the forwarding device toforward data to reach the routing control device; (c) in response toreceipt of a new data stream at the forwarding device, sending a messageto the routing control device on the data link layer port identified in(b), the message instructing the routing control device to determine apath through the plurality of forwarding devices to a destination and toconfigure a subset of the plurality of forwarding devices to forward thenew data stream along the determined path; and (d) forwarding the newdata stream according to configuration commands of the routing controldevice.
 16. The program storage device of claim 15, wherein the sending(c) comprises forwarding the initial packet of the new data streamthrough the control device.
 17. The program storage device of claim 15,the method further comprising (e) forwarding, from the forwardingdevice, the control packet to another forwarding device on a remainingdata link layer port of the forwarding device different from the datalink layer port that the packet was received on in (a).
 18. The programstorage device of claim 17, the method further comprising: (f) at theforwarding device, sending a label to another forwarding device in theplurality of forwarding devices, wherein the label, when attached to apacket received at the forwarding device indicates to the forwardingdevice to forward the packet toward the routing control device on thedata link layer port identified in (b).
 19. The program storage deviceof claim 17, wherein the control packet includes a serial number, andthe method further comprising: (f) determining whether the forwardingdevice has previously forwarded another control packet that is from therouting control device, has the serial number, and indicates anotherestimated time to route a packet from the forwarding device to therouting control device lower than the estimated time in the controlpacket, and wherein the step forwarding (e) occurs when the forwardingdevice is determined to have not previously forwarded the other controlpacket in (f).
 20. The program storage device of claim 15, furthercomprising: (e) sending a message to the routing control device on thedata link layer port identified in (b), the message identifying theforwarding device and other forwarding devices in the plurality offorwarding devices that the forwarding device is connected to, whereinthe routing control device uses the message to build a databaserepresenting a topology of the network and uses the database todetermine the path.
 21. The program storage device of claim 15, whereinthe control packet includes an identifier of the routing control device,and further comprising: (e) receiving, on a data link layer port of theforwarding device, another control packet including an identifier ofanother routing control device; (f) identifying, based at least in parton the data link layer port that received the forwarding device, whichdata link layer port on the forwarding device to forward data to reachthe other routing control device; and (g) in response to receipt of anew data stream at the forwarding device, determining which of therouting control device and the other routing control device to send themessage instructing that the path be configured through the plurality offorwarding devices.